Catch documentation that describes an API we no longer have - #33
Merged
Conversation
Nothing verified that docs.saturday.fit still described the live API, so a
renamed field or a retired endpoint stayed published until a partner's
integration broke against it.
scripts/check-docs-drift.py resolves three things a page can get wrong:
an endpoint against the operations in fuel-backend/api/openapi.yaml, a field
name in a json example against the spec's properties AND the backend's Go
struct tags, and a webhook event name against webhook.AllEventTypes(). The
spec is trustworthy for routes because fuel-backend's openapi-drift gate
already proves its operations and the live mux registrations are the same
set. It is NOT complete for response bodies, so the field check reads the Go
tags too; against the spec alone it reported six real, shipped, correctly
documented nutrition fields as fabrications.
Failures name the endpoint or field, both values, and every file and line.
Found on the first run and fixed here rather than baselined: PATCH
/v1/athletes was documented twice and does not exist. The collection takes
GET and POST; PATCH is only on /v1/athletes/{athlete_id}. The shorthand
"POST and PATCH /v1/athletes" reads as a real endpoint, and a partner
following it gets a 404.
api-drift-baseline.txt carries what is left, four json keys the checker
cannot tell apart from drift: three are MCP client-config settings that
happen to sit in a json fence, one is an open-ended symptom map key. The
list is shrink-only. An entry that stops matching a real finding fails the
gate, because a suppression that suppresses nothing would absorb the next
regression, and the baseline-shrink-only job rejects any PR that appends to
it.
What a green run does not prove is that the numbers are right. Every check
is structural, so a page stating a default of 60 g/hr against code that says
90 passes cleanly. That gap needs values pulled from the code at build time
or golden responses replayed against a live instance; the script header says
so plainly rather than letting a green check imply more than it checked.
Ci-From: linux
Ci-Session: b96c868c-c971-4c89-a902-c735adfd1df9
The two repos are checked out as siblings under $GITHUB_WORKSPACE, matching calculator-crossval.yml, rather than cloning to .. and writing outside the workspace the runner owns. Also records that the closest-known-spelling hint is best effort: it finds carb_pct -> carb_percent, and cannot bridge a wholesale rename, which the finding itself never depends on. Ci-From: linux Ci-Session: b96c868c-c971-4c89-a902-c735adfd1df9
It finds carb_pct -> carb_percent and report_focus -> ai_report_focus. It cannot bridge confidence_score -> prescription_confidence, and no cutoff reaches that rename without also suggesting unrelated fields. The finding never depends on the hint; the file and line numbers carry it. Ci-From: linux Ci-Session: b96c868c-c971-4c89-a902-c735adfd1df9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a check that fails when the documentation and the API disagree. It catches endpoints we document but no longer serve, JSON fields that exist nowhere in the backend, and webhook event names the server will reject.
That last one is not cosmetic: registration stops at the first invalid event name, so one stale name in the docs means a partner who copies our example cannot register any webhook at all.
It already found a real error.
PATCH /v1/athleteswas documented on two pages and does not exist; the collection takes GET and POST, and PATCH is only on a single athlete. Any partner following those pages got a 404. Both pages are corrected here rather than recorded as known debt.Runs in both directions, so a backend change that retires an endpoint goes red while its author is still looking at it, not hours later on someone else's pull request.
What it deliberately does not catch, stated in the workflow itself so a green check is never read as more than it proved: wrong values. A page saying 60 g/hr against code saying 90 passes cleanly. Catching that needs numbers pulled from the code at build time, which is its own project.
🤖 Generated with Claude Code
https://claude.ai/code/session_01H91bd3c6tQoUsP8gBhkpWM